feat(otel-thread-ctx): Node.js OTEP-4947 thread-context writer - #9210
Conversation
Overall package sizeSelf size: 8.04 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🔄 Datadog retried 1 test - 1 passed on retry 🎯 Code Coverage (details) 🔗 Commit SHA: f66073e | Docs | Datadog PR Page | Give us feedback! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9210 +/- ##
========================================
Coverage 98.56% 98.57%
========================================
Files 969 972 +3
Lines 140152 140773 +621
Branches 12529 12050 -479
========================================
+ Hits 138139 138760 +621
Misses 2013 2013
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
BenchmarksBenchmark execution time: 2026-08-12 12:51:19 Comparing candidate commit f66073e in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2309 metrics, 49 unstable metrics.
|
797fe55 to
d608888
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d608888464
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // the result on the span. | ||
| function getCachedWebTags (span) { | ||
| const cached = getCache(span) | ||
| if (cached.resolved) return cached.webTags |
There was a problem hiding this comment.
Invalidate cached misses when parent web tags resolve
When a child span is entered before its HTTP parent receives route/resource tags, this cache stores resolved=true with webTags === undefined for the child. A later dd-trace:span:tags:update on the parent only publishes resolvedCh for the parent span, so re-entering the already-cached child hits this fast path and never re-walks to pick up the now-resolved endpoint; the OTel thread-context record (and wall-profiler context through the shared cache) will keep missing endpoints for common flows where routing tags are set after downstream spans start.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This is a valid issue, although it's been pre-existing for very long time (it existed before in wall.js code before we extracted it into web-tags-cache.js. This is rather non-trivial to fix, we need to either not-cache-negative-answers (extra walks each activation), or track a parent-children reverse map to invalidate on parent transition. Given the complexity, I find it acceptable as a corner case, but I can file a JIRA issue to keep track of it.
There was a problem hiding this comment.
Tracking as https://datadoghq.atlassian.net/browse/PROF-15353 for follow-up
| // wins; subsequent calls are no-ops regardless of their argument. In | ||
| // practice all callers observe the same global ACF state. | ||
| function ensureChannelsActivated (asyncContextFrameEnabled) { | ||
| if (channelsActivated) return |
There was a problem hiding this comment.
Allow later callers to request non-ACF hooks
If DD_TRACE_OTEL_CTX_ENABLED initializes these channels first on an ACF-capable runtime, this flag is set after only the enterWith wrapper is installed. A later wall-profiler start in auto mode with DD_PROFILING_ASYNC_CONTEXT_FRAME_ENABLED=false calls ensureChannelsActivated(false), but returns here before installing the async_hooks.before publisher and run() wrapper that non-ACF profiling relies on, so code-hotspot/endpoint contexts stop updating for that configuration.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This is true, but the scenario is quite unrealistic as it requires the combination of DD_TRACE_OTEL_CTX_ENABLED=1 DD_PROFILING_ASYNC_CONTEXT_FRAME_ENABLED=0 DD_PROFILING_ENABLED=auto. I'll file a JIRA as a low-priority follow-up.
There was a problem hiding this comment.
Filed for eventual follow-up as https://datadoghq.atlassian.net/browse/PROF-15354
…times Addresses Codex review point #4 on #9210: getThreadLocalMetadata would happily return a payload on macOS/Windows or on Node without an active AsyncContextFrame, which would cause libdatadog to advertise a threadlocal block that no writer is actually producing. Gate the function on the same platform + ACF conditions start() already checks, so callers see 'no threadlocal block'.
Addresses Codex review point #2 on #9210: an OTEP-4947 record can only carry the first-written value for each attribute key, and HTTP plugins routinely set 'http.method' up front and add 'http.route' (plus 'resource.name') later once routing has resolved. The writer previously committed the endpoint on first activation, so a request to '/users/:id' was recorded as 'GET' forever. Introduce an isEndpointFinal(tags) heuristic and only write the endpoint when the tag bag looks stable — either 'resource.name' is set, or both 'http.method' and 'http.route' are. Otherwise mark the context as needsEndpoint and re-check on every 'dd-trace:span:tags:update' fire (switched from webTagsCache.resolvedCh, which only fires on presence transitions and would miss content-only updates).
50b56d7 to
eb0ec07
Compare
Addresses Codex review point #2 on #9210: HTTP plugins routinely set 'http.method' up front and add 'http.route' (plus 'resource.name') later once routing has resolved. The writer previously committed the endpoint on first activation, so an out-of-process reader sampling mid-request saw a bare 'GET' as the endpoint for a call to '/users/:id'. OTEP-4947 duplicates are last-wins, so a later appendAttributes would overwrite the interim value for readers that decode the record in full — but a sampling reader can still observe the incomplete value. Introduce an isEndpointFinal(tags) heuristic and only write the endpoint when the tag bag looks stable — either 'resource.name' is set, or both 'http.method' and 'http.route' are. Otherwise mark the context as needsEndpoint and re-check on every 'dd-trace:span:tags:update' fire (switched from webTagsCache.resolvedCh, which only fires on presence transitions and would miss content-only updates).
1a19d66 to
275b832
Compare
| "DD_TRACE_OTEL_CTX_ENABLED": [ | ||
| { | ||
| "implementation": "A", | ||
| "type": "boolean", | ||
| "default": "false" | ||
| } | ||
| ], |
There was a problem hiding this comment.
Fingers crossed we can enable by default (on supported node.js versions) soon ;)
| // Positional attribute layout. The local root span ID stays at index 0 by | ||
| // convention (mirrors libdatadog's libdd-otel-thread-ctx, where | ||
| // `local_root_span_id` is always the first entry in | ||
| // `threadlocal.attribute_key_map`), encoded as a 16-character lowercase | ||
| // hex string. Endpoint, thread name, and thread id follow. Adding more | ||
| // means assigning the next index and updating ATTRIBUTE_KEYS | ||
| // accordingly. |
There was a problem hiding this comment.
Minor: Since we're supplying a threadlocal.attribute_key_map I'm not sure if libdatadog will still prepend the local root span id, might be worth checking
There was a problem hiding this comment.
It does prepend it. At least, the version we built libdatadog-nodejs process_discovery crate still did, if this changes in a later libdatadog release, we should update accordingly. I know there's some talk of retiring local root span id?
There was a problem hiding this comment.
If it worked when you did it, that's fine!
I know there's some talk of retiring local root span id
Yes, it's on my todo list to experiment a bit, but I'll share it loudly if it looks like we're going in that direction so yeah for now let's keep it.
ba55aae to
1ebc38e
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ef6f390d2c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Detaching the finished span's ThreadContext from the current async-context frame only covers that one frame. Sibling frames, and continuations the span scheduled before finishing, inherit the same ThreadContext reference; in ACF mode no `before` hook exists and no storage:enter fires in those frames to overwrite the record, so an out-of-process reader kept seeing the finished span as the active thread context there. Use the new ThreadContext.invalidate() from @datadog/pprof 5.18.0, which marks the record's `valid` byte 0 in place and so drops it out of scope for every frame holding the reference at once. It runs unconditionally: the frame calling span.finish() is not necessarily the frame holding that span's context (a client span finished from a callback where the parent server span is active, or any manual finish() from an unrelated context), and the record dies with the span regardless of who holds it. clearContext() stays gated on the current frame actually being the holder — that call is only about making the record collectable rather than leaving an invalid one attached. Reported by codex on #9210.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 21754042d9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The endpoint name a web-server span's tag bag yields changes as the request
progresses, and the two consumers of that bag disagreed about what to do
about it.
datadog-plugin-next starts its request span with `resource.name: req.method`
and only replaces it with `${req.method} ${page}` once the page is known. The
OTEP-4947 writer treated any `resource.name` as a settled value, so a Next
request's record committed a bare `GET` on first activation and ignored the
resolved route forever. The same hole existed for `http.route` arriving on a
span whose `resource.name` was still the bare method, because
endpointNameFromTags prefers `resource.name`: what got written was the
placeholder even though the route was known. Finality is now decided on the
computed value rather than on which tags are present, which covers both.
That predicate moves to profiling/webspan-utils.js next to
endpointNameFromTags, since the wall profiler needs it too: it resolves
endpoint labels lazily at serialization time and so never saw the problem,
but the fallback endpoint it snapshots mid-request for when the tag bag is
unreadable later is never refreshed once set, and could be pinned to a bare
`GET`.
Records built before the endpoint settles also have to be filled in
afterwards, and only the request span's own record was. A descendant resolves
to its nearest web-server ancestor's tag bag and gets its own record with its
own endpoint copy, but the tags update carrying the route is published for the
ancestor, which cannot enumerate its descendants — so a span entered during
the deferral window never got an endpoint at all. web-tags-cache now announces
the moment a request's endpoint settles on endpointResolvedCh, and the writer
keeps the records waiting on that announcement grouped by the tag bag they are
waiting on, filling in every one of them at once. Records whose span finished
in the meantime are skipped: onSpanFinished already invalidated them.
Driving that off the cache's transition channels rather than off
`dd-trace:span:tags:update` directly leaves one subscriber on that hot channel
instead of two, and removes the writer's ordering dependency on the cache
having processed an update before it reads the cache.
web-tags-cache had no spec of its own despite now having two consumers; this
adds one pinning both transitions and the activation refcount.
Reported by codex on #9210.
…prof
Both failures this prevents would have surfaced from inside a diagnostic-channel
subscriber, which runs inline with the tracer's hot path, so the exception would
have landed in application code.
The compatibility check gated the otelThreadCtx namespace but not the
ThreadContext methods the writer calls on it. An older or overridden
@datadog/pprof exposing the namespace without invalidate() would pass the gate
and then throw out of the span-finish path and up through DatadogSpan#finish()
the first time an activated span finished. The check now covers appendAttributes,
enter and invalidate as well, and names the missing member in the warning.
@datadog/pprof also decides whether AsyncContextFrame is available by inspecting
process.execArgv, and throws from enter() when it concludes it is not. That
disagrees with the feature detection behind isACFActive whenever the flag reached
Node by another route, which is reachable today:
$ NODE_OPTIONS=--experimental-async-context-frame node -e '...' # Node 22.23.2
{"isACFActive":true,"execArgv":[],"pprofWouldThrow":true}
Node 22 and 23 accept the flag in NODE_OPTIONS (Node 24 rejects it, and does not
need it), and a worker thread created with an explicit execArgv loses it too. In
those processes the first span activation would have thrown from enter() into
whatever code triggered it. start() now installs and detaches one throwaway
context up front and declines to start if that fails, so an unusable pprof costs
a warning instead of an application-visible exception.
The execArgv inference is worth replacing with feature detection upstream in
pprof-nodejs; this check is what keeps a dd-trace release safe against any
version that still has it.
Reported by codex on #9210.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bea0d4f084
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const startedSpans = getStartedSpans(spanContext) | ||
| const rootContext = startedSpans.length ? startedSpans[0].context() : spanContext | ||
| // Only write the endpoint when its value has settled; otherwise leave a hole | ||
| // and wait for webTagsCache to announce that it has. | ||
| const webTags = webTagsCache.getCachedWebTags(span) |
There was a problem hiding this comment.
Preserve trace ancestry after partial flush
For traces that hit partial flush before all descendants finish, span_processor.js prunes context._trace.started down to only still-active spans, so the original local root and any finished web-server ancestor can disappear while child spans keep running. In that state this path chooses the first remaining active span as datadog.local_root_span_id and webTagsCache.getCachedWebTags() can no longer walk back to the request span, causing OTEP records for the rest of a large/long-lived request to carry the wrong root id and lose the endpoint; the writer needs ancestry/root data that survives partial flush.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This is true, and the mechanism is exactly as described: _erase() in span_processor.js ends with trace.started = active, where active collects only spans whose _duration is still undefined. So when a flush happens while the local root has already finished, the root drops out, started[0] becomes the
oldest span still running, and a finished web-server ancestor stops being reachable for web-tags-cache's parent-chain walk.
It isn't specific to this writer, though. started[0]-is-the-local-root is a repo-wide convention shared by priority_sampler.js, span_format.js, span_context.js, event_plugins/event.js, the wall profiler's
local-root-span-id label, profiler.js's endpoint counting, and web-tags-cache.js's parent walk. (Yeah, profiler is a bit overrepresented.) Partial flush skews all of them the same way, in code that ships today; this writer inherits the behavior rather than introducing it.
That's also why I'd rather not fix it locally here. A writer-local workaround (memoizing the first root id seen per trace, say) would make OTEP records disagree with the wall profiler's labels about local_root_span_id for the same span, and two profilers reporting different roots is worse than both sharing one skew.
Two things bound the impact meanwhile: it needs a trace past DD_TRACE_PARTIAL_FLUSH_MIN_SPANS (default 1000) and a root that finished while descendants keep running; and web-tags-cache memoizes the resolved tag bag per span, holding the bag itself rather than re-walking, so any span that resolved its endpoint before the flush keeps it. Only spans first activated after a partial flush are affected.
The real fix is a local-root reference the tracer core maintains across flushes, applied to all of the consumers above at once, that's tracer-core work, and its own change. getStartedSpans in profiling/webspan-utils.js now documents the limitation at the one point every consumer reads, so the next person to touch it doesn't have to rediscover it.
Move the dd-trace storage diagnostics channels (storage:enter, storage:before, span:finish, span:tags:update), the legacy-storage enterWith/run shimmer, and the getActiveSpan helper out of profiling/profilers/wall.js into a new packages/dd-trace/src/storage-channels.js. No behavior change for the wall profiler. The extraction is to let a forthcoming OTEP-4947 thread-context writer reuse the same channel infrastructure without duplicating the shimmer or pulling in the profiler module. Adjust wall.spec.js's proxyquire stubs to replace the new module instead of mocking datadog-core directly.
bea0d4f to
63161b5
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 63161b5c04
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The wall profiler and the OTEP-4947 thread-context will both walk each span's started-spans chain to find the nearest web-server ancestor, each caching the answer under its own Symbol. Move the walk and cache into packages/dd-trace/src/web-tags-cache.js. - getCachedWebTags(span): lazy parent-chain walk, cached on a shared Symbol. - onSpanTagsUpdated(span): call from a tagsUpdate subscriber; if the walk previously came up empty and the span is now a web-server span, promote its tags into the cache. Returns true iff the cache transitioned from undefined to a real value — a signal to consumers that they should snapshot the new value into whatever they built while the answer was undefined. wall.js still snapshots webTags into its per-sample ProfilingContext because label generation reads it from the sample-context ref (not from the span); its tagsUpdate handler now refreshes that snapshot only when the shared cache signals a transition. No functional change. CODEOWNERS gets the new file scoped to @DataDog/profiling-js.
New module packages/dd-trace/src/otel-thread-ctx.js that mirrors the
active trace ID, span ID and current endpoint into a thread-local
OTEP-4947 record. An out-of-process eBPF reader discovers the record
via the otel_thread_ctx_nodejs_v1 TLS symbol exported by the
@datadog/pprof addon.
Highlights:
- Gates on Linux + AsyncContextFrame (Node 24+ default, or Node 22/23
with --experimental-async-context-frame). isACFActive from
datadog-core/src/storage is the single source of truth for that
check.
- One ThreadContext is allocated per span the first time it's
activated and cached on the span via a Symbol slot. Re-entries in
any async-context frame re-install the same wrap via setContext;
`getContext() === cachedContext` is the JS-reference identity
check that replaces any byte-level comparison (same allocation-
churn fix as the wall profiler in dd-trace-js#8638).
- The record always carries the local root span ID (16-char hex,
index 0 by libdatadog convention), thread name and thread id
(stable per-thread, computed once at module load). The endpoint
attribute is appended in place when a web-server ancestor is
discovered via the span:tags:update channel.
- On span:finish, if the writer's record currently belongs to the
finishing span, setContext(undefined) is called so an out-of-
process reader doesn't keep seeing a finished span as the active
thread context (matters in enterWith-style sticky activation).
Activation gate:
- DD_TRACE_OTEL_CTX_ENABLED (boolean, default false) is registered
in supported-configurations.json as implementation A, exposed as
config.DD_TRACE_OTEL_CTX_ENABLED. proxy.js calls
otel-thread-ctx.start() iff the flag is set; the module itself
starts in a no-op state otherwise.
Tests + docker rig:
- packages/dd-trace/test/otel-thread-ctx.spec.js covers start()
gating, enter/skip behavior, span-drift round-trip, span-finish,
and the late-tags append path with the wire-record attribute
layout (15 cases).
- scripts/docker/{Dockerfile,run-otel-thread-ctx-spec.sh} build a
node:24-bookworm image and run the spec against the locally built
sibling pprof-nodejs; driven by `npm run test:docker:otel-thread-ctx`.
Forthcoming follow-up: publishing the corresponding
threadlocal.attribute_key_map via process discovery, which requires
bumping @DataDog/libdatadog.
Sets up the OTEP-4719 process context so an out-of-process reader can decode
the on-the-wire records the thread-context writer emits. The metadata is
published through libdatadog-nodejs's process-discovery napi crate: bumped
here to 0.12.0, which exposes the ThreadLocalMetadata substruct with the full
'threadlocal.*' block (attribute key map, schema-version string, and extra
KeyValues for reader-side layout constants).
The pieces:
- Add getThreadLocalMetadata() in otel-thread-ctx.js. Pulls the process-context
snapshot from @datadog/pprof (its otelThreadCtx.getProcessContextAttributes
is the source of truth for the schema-version string and V8 layout constants
the reader needs) and reshapes it into the napi ThreadLocalMetadata form:
{ attributeKeys, schemaVersion, extraAttributes: [{ key, intValue|stringValue }] }.
Returns undefined when @datadog/pprof isn't installed or doesn't expose the
helper.
- Wire tracer_metadata.js to pass the substruct (or undefined) as the last
positional arg to processDiscovery.TracerMetadata(...), replacing the flat
threadlocalAttributeKeys array. Gated on the same DD_TRACE_OTEL_CTX_ENABLED
flag that activates the writer.
- Bump the @DataDog/libdatadog optionalDependency from 0.10.0 to 0.12.0.
The spec is technically already picked up by the top-level 'packages/dd-trace/test/*.spec.js' glob in test:trace:core:ci (apm-capabilities.yml), but the file is profiling-team-owned per CODEOWNERS. Adding a dedicated step in profiling.yml gives the owning team direct failure visibility in their own workflow.
63161b5 to
f66073e
Compare
The shared web-tags cache records "no web-server ancestor" for a span and never revisits it, but that answer expires: plugins set `span.type` after creating the span — TracingPlugin.startSpan activates it before addRequestTags runs — so a child created in that window walks past an ancestor that is about to become a web-server span, and caches a miss for a chain that is about to have one. Promotion can't find those descendants, since the walk only goes upwards. So a promotion now bumps a generation counter, and an empty answer older than the counter is walked again on the next lookup. Resolved answers are untouched (the cached bag is the ancestor's live tag object), and a span with no parent is stamped as permanently empty, since only its own promotion could change it and onTagsUpdate already handles that. When a re-walk turns an empty answer into a real one, the cache publishes resolvedCh for that span, the same announcement a promoted span gets, so a consumer doesn't have to care which way the ancestry appeared. The counter lives on the trace, not in the module. A promotion can only invalidate empty answers within its own trace, because the walk follows `_parentId` through `_trace.started` and never leaves it — while a process-global counter would have every HTTP request's promotion invalidate empty answers in unrelated traces, making a long-lived non-web span re-walk its chain once per request served elsewhere, from the storage-enter path. Invalidation alone fixes nothing, because both consumers only ask the cache while building their per-span state, and the spans this affects already have theirs built. So each now asks again for state built from an empty answer: - the OTEP-4947 writer re-checks on re-entry when a record was built with no web-server ancestor, and attaches the endpoint or enlists the record for the request's endpoint announcement. Guarded so a re-entrant announcement from inside the lookup can't append the endpoint twice. - the wall profiler re-checks in #getProfilingContext when the snapshot it holds has no webTags, since #spanTagsUpdated only fires for a span promoted itself, never for descendants that walked past it beforehand. Both re-checks cost two property reads plus, at most, the cache's own generation compare — a walk only happens when a promotion in that trace has actually invalidated something. Two test-harness inaccuracies are fixed along the way, both of which had been hiding behaviour rather than testing it: web-tags-cache.spec.js's makeSpan returned a fresh object from context() per call, so spying on it counted calls on a throwaway and the "walks the parent chain once" assertion held vacuously; and wall.spec.js's makeChildSpan gave the child its own _trace object, so parent and child were in different traces, which no real trace chunk is. Reported by codex on #9210 and #9805.
Mirrors the active trace ID, span ID, local root span ID and current endpoint into a thread-local OTEP-4947 record, so an out-of-process eBPF reader can attribute samples without going through the tracer. The record is discovered via the otel_thread_ctx_nodejs_v1 TLS symbol exported by the @datadog/pprof addon, and decoded using an OTEP-4719 process context published through libdatadog's process discovery (threadlocal.* attribute key map, schema version, and the V8 layout constants a reader needs to walk into the record). Off by default, behind DD_TRACE_OTEL_CTX_ENABLED. Requires Linux and an active AsyncContextFrame (default from Node 24, opt-in on 22/23), and refuses to start unless the installed @datadog/pprof exposes every API member the writer calls and a context can actually be installed in this process — an unusable pprof costs a log line rather than an exception thrown from a hot-path diagnostic channel subscriber. One ThreadContext is built per span on first activation and cached on the span, so re-entry in another async-context frame re-installs the same reference rather than allocating. On span finish the record is invalidated in place, which drops it out of scope for every frame that inherited it — sibling frames and continuations the span scheduled before finishing — since no later storage event reaches those. The endpoint is held back until its value settles: plugins publish interim routing tags, and datadog-plugin-next seeds resource.name with the bare request method, so publishing early would leave a reader attributing samples to "GET". Once a request's endpoint resolves it is appended to every record built under that request, the request span's own and each descendant's.
Mirrors the active trace ID, span ID, local root span ID and current endpoint into a thread-local OTEP-4947 record, so an out-of-process eBPF reader can attribute samples without going through the tracer. The record is discovered via the otel_thread_ctx_nodejs_v1 TLS symbol exported by the @datadog/pprof addon, and decoded using an OTEP-4719 process context published through libdatadog's process discovery (threadlocal.* attribute key map, schema version, and the V8 layout constants a reader needs to walk into the record). Off by default, behind DD_TRACE_OTEL_CTX_ENABLED. Requires Linux and an active AsyncContextFrame (default from Node 24, opt-in on 22/23), and refuses to start unless the installed @datadog/pprof exposes every API member the writer calls and a context can actually be installed in this process — an unusable pprof costs a log line rather than an exception thrown from a hot-path diagnostic channel subscriber. One ThreadContext is built per span on first activation and cached on the span, so re-entry in another async-context frame re-installs the same reference rather than allocating. On span finish the record is invalidated in place, which drops it out of scope for every frame that inherited it — sibling frames and continuations the span scheduled before finishing — since no later storage event reaches those. The endpoint is held back until its value settles: plugins publish interim routing tags, and datadog-plugin-next seeds resource.name with the bare request method, so publishing early would leave a reader attributing samples to "GET". Once a request's endpoint resolves it is appended to every record built under that request, the request span's own and each descendant's.
Mirrors the active trace ID, span ID, local root span ID and current endpoint into a thread-local OTEP-4947 record, so an out-of-process eBPF reader can attribute samples without going through the tracer. The record is discovered via the otel_thread_ctx_nodejs_v1 TLS symbol exported by the @datadog/pprof addon, and decoded using an OTEP-4719 process context published through libdatadog's process discovery (threadlocal.* attribute key map, schema version, and the V8 layout constants a reader needs to walk into the record). Off by default, behind DD_TRACE_OTEL_CTX_ENABLED. Requires Linux and an active AsyncContextFrame (default from Node 24, opt-in on 22/23), and refuses to start unless the installed @datadog/pprof exposes every API member the writer calls and a context can actually be installed in this process — an unusable pprof costs a log line rather than an exception thrown from a hot-path diagnostic channel subscriber. One ThreadContext is built per span on first activation and cached on the span, so re-entry in another async-context frame re-installs the same reference rather than allocating. On span finish the record is invalidated in place, which drops it out of scope for every frame that inherited it — sibling frames and continuations the span scheduled before finishing — since no later storage event reaches those. The endpoint is held back until its value settles: plugins publish interim routing tags, and datadog-plugin-next seeds resource.name with the bare request method, so publishing early would leave a reader attributing samples to "GET". Once a request's endpoint resolves it is appended to every record built under that request, the request span's own and each descendant's.
Mirrors the active trace ID, span ID, local root span ID and current endpoint into a thread-local OTEP-4947 record, so an out-of-process eBPF reader can attribute samples without going through the tracer. The record is discovered via the otel_thread_ctx_nodejs_v1 TLS symbol exported by the @datadog/pprof addon, and decoded using an OTEP-4719 process context published through libdatadog's process discovery (threadlocal.* attribute key map, schema version, and the V8 layout constants a reader needs to walk into the record). Off by default, behind DD_TRACE_OTEL_CTX_ENABLED. Requires Linux and an active AsyncContextFrame (default from Node 24, opt-in on 22/23), and refuses to start unless the installed @datadog/pprof exposes every API member the writer calls and a context can actually be installed in this process — an unusable pprof costs a log line rather than an exception thrown from a hot-path diagnostic channel subscriber. One ThreadContext is built per span on first activation and cached on the span, so re-entry in another async-context frame re-installs the same reference rather than allocating. On span finish the record is invalidated in place, which drops it out of scope for every frame that inherited it — sibling frames and continuations the span scheduled before finishing — since no later storage event reaches those. The endpoint is held back until its value settles: plugins publish interim routing tags, and datadog-plugin-next seeds resource.name with the bare request method, so publishing early would leave a reader attributing samples to "GET". Once a request's endpoint resolves it is appended to every record built under that request, the request span's own and each descendant's.
Summary
Adds a Node.js writer for the OpenTelemetry Thread Local Context Record (OTEP-4947), letting out-of-process readers (typically eBPF profilers) sample the active trace/span ID and a small attribute payload with no cooperation from the tracer at read time. Gated behind
DD_TRACE_OTEL_CTX_ENABLED(default off).The writer itself lives in
@datadog/pprof(5.16.0+); this branch wires it into the tracer and publishes the accompanying OTEP-4719 process context via libdatadog-nodejs (0.12.1+).Relevant PRs in other repos that this PR builds upon:
A note on future work: unifying CPU profiler context and OTel thread context
There's lots of similarities in span-related context management between the new writer in
otel-thread-ctx.jsand the in-process CPU profiler inprofiler/wall.js. Two of the commits in this PR extract common functionality (storage-channels.jsandweb-tags-cache.js) for both, these are elaborated more on below. It would be possible to implement this so thatwall.jsno longer maintains its own context data, but always uses the OTel context instead. Java and PHP profilers already do this. For us, the biggest blocker is that OTel context record relies on Async Context Frame, and the CPU profiler still needs to support Node.js 22-23 where it's off by default, so we can't unify before our lowest supported version is 24 where ACF is on by default.It also has some extra runtime cost, but that's only a minor aspect.
wall.jscurrently establishes its context very cheaply by only retaining references to span-related objects, and deferring string conversion until profile serialization, so it only happens for those contexts (~6k of them/minute) that were captured with samples. In contrast, utf-encoded string data need to be written into the OTel thread context immediately for each created span, since we don't know when it will be captured by an external eBPF reader.We will likely still do the unification, especially if both are enabled by default so the OTel context record is generated anyhow. We can either do the unification during dd-trace-js 6.x cycle but then
wall.jswill be more complex as it'll have to handle both kinds of contexts, or defer until 7.x next year when the minimum supported Node.js version will be 24 so we can dropwall.jsown context and just use OTel.What's in this branch (commit by commit):
Bump @datadog/libdatadog to 0.12.1libdatadog 0.12.1 is a bug-fix bump over 0.12.0 which introduced some functionality we need.
Extract storage-channels module from wall profilerPull the
dd-trace:storage:enter/:before/dd-trace:span:finish/:tags:updatediagnostic-channel wiring out of the wall profiler into a sharedpackages/dd-trace/src/storage-channels.js, so both the wall profiler and the new thread-context writer can subscribe to the same normalized activation stream. No functional change to the wall profiler.Extract shared web-tags cache from wall profilerThe OTel thread context writer will need to walk the started-spans chain per span to find the nearest web-server ancestor, just like wall profiler does. For thi reason, we extract the functionality into
packages/dd-trace/src/web-tags-cache.js: a single Symbol on the span, one lazy walk per span, and add-trace:web-tags:resolveddiagnostics channel that fires once per span at the moment a previously-empty answer transitions to a real value viadd-trace:span:tags:update.Add OTEP-4947 thread context writerAfter the first three preparatory commits, this is the actual new functionality.
packages/dd-trace/src/otel-thread-ctx.js: the writer. Subscribes to storage-channels; on eachstorage:enter, builds (or reuses) aThreadContextfrom@datadog/pprof.otelThreadCtxfor the active span, populates trace/span IDs plus a positional attribute array (index 0 =datadog.local_root_span_id, thendatadog.trace_endpointfor web-server spans,datadog.thread_name,datadog.thread_id), and installs it viacontext.enter(). Handles span-drift (re-installs the same cached context when we switch spans and back) and span-finish (clears the writer if the record is still ours to avoid leaking stale state past enterWith-style activation). Late endpoint discovery is handled via the shared web-tags cache (see below): the writer subscribes towebTagsCache.resolvedChand appends the endpoint attribute in place when the shared cache signals a transition.packages/dd-trace/src/proxy.js: gatedrequire('./otel-thread-ctx').start()after profiler init.DD_TRACE_OTEL_CTX_ENABLEDadded tosupported-configurations.json; the generated.d.tspicks it up.scripts/docker/: atest:docker:otel-thread-ctxharness that builds insidenode:24-bookworm(the writer is Linux+AsyncContextFrame-only; macOS dev machines fall through to the harness).packages/dd-trace/test/otel-thread-ctx.spec.js, 20 cases):start()gate matrix, on-enter build/skip/drift, span-finish clear-vs-leave, tags-update endpoint append, and the process-context helper.Publish OTEP-4947 process-context metadata via process discoveryFor OTel thread context record to work correctly, information also needs to be published in the process context.
Sets up the OTEP-4719 process context so an out-of-process reader can decode the on-the-wire records. Adds
getThreadLocalMetadata()inotel-thread-ctx.js— pulls the snapshot from@datadog/pprof.otelThreadCtx.getProcessContextAttributes(schema-version string, attribute key map, V8 layout constants) and reshapes it into the napiThreadLocalMetadataform.tracer_metadata.jspasses it as the last positional arg toprocessDiscovery.TracerMetadata(...). Returnsundefinedif pprof is missing → publishes without a threadlocal block.Platform / runtime scope
--experimental-async-context-frameon 22/23). On any other environment,start()logs and returnsfalse— no runtime cost.@datadog/pprofis an optionalDependency; if it isn't installed, the writer stays inert and no threadlocal block is published in the process context.Test plan
yarn test:otel-thread-ctx— 20/20 passyarn mocha --timeout 60000 packages/dd-trace/test/tracer_metadata.spec.js— 11/11 passyarn mocha --timeout 60000 packages/dd-trace/test/profiling/profilers/wall.spec.js— 32/32 pass (unchanged from master after the shared-cache extraction)Jira: PROF-15220